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Abstract 

This  paper  describes  the  challenging  task  of  testing 
the  most  complex,  expensive,  technically  evolving,  and 
politically  sensitive  missile  system  to  be  developed  to 
date.  The  National  Missile  Defense  (NMD)  System 
comprises  radars,  interceptors,  space-based  sensors,  and 
battle  management,  command,  control,  and  communi¬ 
cations  (BMC3)  elements.  While  the  integration  of 
these  many  complex  elements  into  a  functional  system 
is  no  small  achievement,  the  performance  verification 
of  this  system  is  an  equally  challenging  task. 

The  NMD  Test  Program  is  structured  to  provide 
demonstrated  evidence  of  progress  toward  verifying 
system-level  functional  capability.  The  high  cost  and 
other  constraints  of  flight  tests  require  complementary 
testing  and  evaluation  of  the  NMD  System  capability 
by  means  of  integrated  ground  testing  and  simulation. 
The  recent  development  by  the  System  Engineer  of  a' 
System  Evaluation  Plan  has  provided  a  key  link  from 
the  desired  system  performance,  as  dictated  by  the 
NMD  System  requirements,  to  the  data  items  that  must 
be  collected  during  each  test.  Simulation,  integrated 
ground  testing,  risk  reduction  flights,  and  flight  testing 
are  test  components  whose  results  will  be  used  to  assess 
system-level  performance  of  the  progressively  maturing 
NMD  elements  and  their  associated  test  articles. 

This  paper  describes  the  test  components  of  the 
NMD  test  program,  their  importance  to  the  test  pro¬ 
gram,  relationships  among  the  components,  and  their 
contributions  to  NMD  test  and  evaluation  which  form 
the  basis  for  the  System  Engineer  to  evaluate  and  verify 
NMD  System  performance. 

NMD  Testing  Challenge 

Planning  and  defining  a  test  program  for  the 
nation’s  ballistic  missile  defense  is  a  challenge  in  which 
government  and  industry  have  teamed  for  over  30 
years.  The  NMD  has  evolved  from  a  Department  of 
Defense  (DoD)  major  ballistic  defense  acquisition  pro¬ 
gram — Global  Protection  Against  Limited  Strikes 
(GPALS) — in  1993,  to  an  element-focused  Technology 
Readiness  Program,  to  the  current  Deployment  Readi¬ 
ness  Program  initiated  on  9  April  1996.  Also  known  as 
“3  plus  3,”  the  program  faces  a  challenge  of  compress¬ 


ing  to  6  years  the  normal  10-  to  12-year  acquisition 
cycle  of  a  typical  DoD  program.  The  program  will 
develop,  integrate,  and  test  a  limited  system  capability 
in  3  years  that  could  be  deployed  in  a  number  of  system 
configurations  during  the  following  3  years.  If  a  deci¬ 
sion  to  deploy  the  system  is  not  made  after  the  3-year 
development,  the  program  would  continue  to  maintain  a 
capability  to  deploy  an  NMD  System  within  3  years 
after  a  deployment  decision  is  made  while  improving 
the  performance  and  robustness  of  the  system. 

Elaborate  planning  is  required  to  accommodate 
various  challenges  faced  in  conducting  NMD  tests  at 
various  levels.  Some  of  these  challenges  are  as  follows: 
1)  The  threat  and  environment  against  which  the 
deployed  NMD  System  is  designed  to  operate  are 
unavailable.  The  targets  used  in  these  flight  tests  are 
not  the  true  representations  of  the  actual  wartime  threat. 
Therefore,  to  test  the  specified  performance,  simula¬ 
tions  are  extensively  used.  2)  Elements  mature  at  dif¬ 
ferent  rates  during  their  development  cycle.  This 
requires  a  trade  between  the  test  schedule,  the  use  of 
test  drivers,  and  the  tailoring  of  test  objectives  to 
accommodate  the  maturity  levels  of  various  elements. 
3)  The  ground  and  flight  tests  are  scheduled  to  com¬ 
plement  each  other.  A  pre-mission  ground  test  supports 
the  following  flight  test,  and  a  post-mission  ground  test 
uses  the  data  from  the  preceding  flight  test  to  verify 
various  algorithms.  Any  schedule  slip  in  delivery (ies), 
lack  of  specified  capabilities  in  one  or  more  elements, 
or  any  other  interruption  in  this  cycle  breaks  the  pre¬ 
planned  schedule  of  test  events  and  requires  a  complete 
reassessment  of  the  test  program.  4)  The  Compliance 
Review  Group  (CRG)  is  briefed  well  in  advance  on  the 
treaty  compliance  of  the  flight  tests.  5)  Test  facilities 
are  planned  to  ensure  that  full  environmental  impacts 
must  be  considered  and  approved. 

Svstem/Element  Overview 

The  initial  architecture  for  the  NMD  System  could 
be  in  place  by  2003  if  a  deployment  is  mandated  in 
FYOO.  This  system  will  meet  the  threshold  values  of 
the  User’s  operational  requirements  and  provide  high 
levels  of  effectiveness  against  a  limited  threat  com¬ 
prising  a  few  simple  warheads.  The  NMD  System 
architecture  will  comprise  several  major  elements 
(Figure  1).  The  Ground  Based  Interceptor  (GBI)  is  a 
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Figure  1.  NMD  System  Elements 


kinetic  energy  exoatmospheric  interceptor  with  long 
flyout  range  to  provide  multiple  engagement  capability 
in  the  midcourse  phase  of  a  threat  Intercontinental  Bal¬ 
listic  Missile  (ICBM)  flight  trajectory.  The  Ground 
Based  Radar  (GBR),  or  site  radar,  is  an  X-band  radar 
that  would  provide  surveillance,  tracking,  discrimina¬ 
tion,  and  kill  assessment  data.  Upgraded  Early  Warn¬ 
ing  Radars  (UEWR)  extend  the  NMD  battlespace  by 
providing  target  detection  earlier  than  a  single-site 
GBR,  The  existing  Early  Warning  Radars  (EWRs) 
could  be  upgraded  to  provide  this  enhanced  surveil¬ 
lance  and  tracking  data.  The  X-Band  Radars  (XBRs) 
provide  forward-based,  high-resolution  data  from  ear¬ 
lier  phases  of  an  ICBM’s  trajectory  for  surveillance, 
tracking,  discrimination,  and  kill  assessment.  The 
Defense  Support  Program  (DSP)/Space  Based  Infrared 
System  (SBIRS)  High  Component  satellites  maintain 
continuous  global  surveillance  to  detect  all  ballistic 
missile  launches  and  provide  early  warning  and  booster 
track  data.  The  SBIRS  Low  Component  satellites  will 
be  designed  to  provide  boost  and  mid-course  tracking, 
discrimination,  and  hit  assessment  data.  The  Battle 
Management/Command,  Control,  and  Communications 
(BMC3)  element  provides  essential  planning,  tasking, 
and  controlling  operations  of  assigned  assets.  It  also 
provides  communications  network  management,  In- 
Flight  Interceptor  Communications  System  (IFICS), 
and  local  and  wide  area  communications. 

Evolution  of  Maturity  and  Integration 

A  progressive  plan  is  laid  with  a  build-up  of  sys¬ 
tem-level  integrated  testing  using  simulations,  ground 


tests,  and  flight  tests  with  threat-representative  targets. 
NMD  System  maturity  and  capability  evolve  through 
an  incremental  and  logical  build-up  sequence  of  critical 
interfaces  and  system  performance.  The  incremental 
build-up  of  tested  capability  integrates  element  devel¬ 
opment  items  (representations)  using  early  rep¬ 
resentations  for  unavailable  elements.  Element  test 
article  representations  evolve  from  models  to  functional 
representations,  to  actual  hardware  and  software 
processors  that  are  used  for  element  and  system 
physical  integration  and  test. 

Verification  of  Technical  Performance 

System  assessment  begins  with  functional  testing 
of  integrated  element  models  and  their  test  drivers  to 
check  out  algorithms  and  validate  the  concepts. 
Assessment,  then,  progresses  to  real-time  simulation  to 
evaluate  timing  and  Human-In-Control  (HIC)  aspects. 
Physical  and  logical  interfaces  are  tested  using  matur¬ 
ing  element  hardware  and  software  as  available,  and 
integrated  either  in  a  contractor  laboratory  or  through 
use  of  a  wide  area  network  distributed  test.  System- 
level  testing  capitalizes  on  planned  element  tests 
through  collecting  the  appropriate  data  early.  This  data 
is  used,  subsequently,  in  effectively  planning  and 
conducting  the  dedicated  system  tests  to  verify  system 
performance.  These  tests  have  deployable  element  rep¬ 
resentations  and  are  tested  in  configurations  and  sce¬ 
narios  supporting  technical  and  operational  perform¬ 
ance  requirements.  Integrated  flight  tests  (IFTs)  con¬ 
firm  the  performance  of  actual  elements  in  a  live-range 
setting  using  a  limited  threat  scenario,  while  ground 
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tests  extend  the  performance  demonstration  through 
element  hardware-in-the-loop  (HWIL)  representations 
and  to  full  operational  scenarios.  Figure  2  depicts  the 
four  components  of  the  NMD  Test  Program,  i.e.,  simu¬ 
lation,  ground  testing,  flight  testing,  and  risk  reduction 
flights  (RRFs).  Before  discussing  these  components, 
we  will  provide  background  regarding  the  unique  NMD 
testing  limitations  and  constraints  and  an  overview  of 
the  test  planner/system  engineering  process. 

Limitations  and  Constraints  For  Testing 

A  brief  discussion  will  reveal  why  considerations 
for  test  planning  and  conduct  are  made  to  mitigate  vari¬ 
ous  limitations  and  constraints.  Replicating  total  sys¬ 
tem  realism  and  performance  is  bounded  by  the  unique 
characteristics  of  the  NMD  elements  and  the  system  test 
infrastructure.  The  following  descriptions  will  further 
exemplify  the  characteristics  of  the  NMD  test  program. 
Integrated  testing  depends  on  availability  of  element 
test  articles,  their  functional  capabilities  and  maturity 
levels,  and  resources.  Due  to  the  high  cost  and  com¬ 
plexity  of  flight  testing,  resources  are  planned  to  maxi¬ 
mize  their  value,  and  integration  opportunities  are 
planned  to  minimize  cost  and  schedule  impacts. 
Because  of  budget  and  time  constraints,  targets  are  con¬ 
structed  to  be  as  representative  as  practical  of  the 
threats  postulated  for  the  deployment  time  frame.  Con¬ 
sequently,  to  physically  test  all  engagement  variants 
against  actual  threats,  a  major  focus  must  be  simulation 
and  emulation  development.  Due  to  the  nature  of  bal¬ 
listic  missile  defense,  environmental  pollution  preven¬ 
tion  and  treaty  compliance  must  be  planned  for  in  all 


tests.  Another  consideration  recently  realized  is  that 
since  the  full  complement  of  HWIL  and  software-in- 
the-loop  (SWIL)  test  article  quantities  is  not  affordable, 
testing  full  capability  architectures  is  limited.  There¬ 
fore,  analysis  and  simulation  are  used  to  augment 
testing. 

Test  Program  Overview 

The  test  program  is  flexible  in  order  to  embrace 
each  step  or  degree  of  system  performance  as  the  ele¬ 
ments  mature.  The  high  cost  and  the  constraints  of 
flight  testing  place  the  burden  of  evaluating  the  NMD 
System  capability  on  the  ground  tests  and  simulations. 
As  element  developers  produce  test  articles  and  pro¬ 
totypes  on  their  developmental  schedules,  representa¬ 
tions  with  increasing  levels  of  fidelity  are  used  for 
integrated  system  testing.  Modeling  and  simulation. 
Integrated  Ground  Test  (IGT),  IFT,  and  RRF  results  are 
used  to  assess  system-level  performance  as  the  ele¬ 
ments  and  their  associated  test  articles  mature.  The 
NMD  System  test  program  achieves  flexibility  by  pro¬ 
viding  a  system-level  assessment  of  integrated  element 
performance  at  progressive  levels  of  maturity. 

As  system  integrated  tests  are  conducted,  the 
results  are  evaluated  against  test  predictions  and  per¬ 
formance  measures  derived  from  the  critical  technical 
parameters  (CTPs)  and  critical  operational  issues 
(COIs).  The  evaluated  integrated  results  are  combined 
with  element  test  results  to  adjust  the  plans  for  future 
tests.  Results  of  these  element  and  system  tests  support 
periodic  appraisals  of  demonstrated  capability  and 
confidence  measures  by  the  NMD  System  Engineer. 
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Figure  2.  Components  of  the  NMD  Test  Program 
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The  System  Evaluation  Plan  (SEP),  developed  by 
the  System  Engineer,  delineates  the  methodology  for 
evaluating  the  performance  of  the  NMD  System  in 
preparation  for  the  Deployment  Readiness  Review 
(DRR)  in  FYOO.  The  SEP  is  derived  from  the  Capa¬ 
bility  1  (Cl)  System  Requirements  Document  (SRD) 
and  defines  technical  performance  measures  (TPMs) 
that  summarize  system  performance  in  eight  critical 
areas.  These  TPMs  and  their  evaluation  are  supported 
by  101  subordinate,  related  evaluation  parameters 
(EPs).  These  EPs  have  been  selected  from  the  633 
“Perform  NMD”  requirements  in  the  Cl  SRD  based  on 
how  sensitive  the  negation  probability  would  be  to  the 
parameters’  variation  and  the  level  of  technical  concern 
within  the  community.  For  each  of  these  EPs,  the  SEP 
identifies  test  data  categories  and  specific  test  data 
items  that  must  be  collected  for  the  System  Engineer’s 
performance  analysis. 

These  specific  data  items  are  further  decomposed 
by  the  test  planners  to  identify  test  data  requirements 
for  each  integrated  system  level  test.  Coordination  of 
these  requirements  with  the  element  developers  is  vital 
to  assure  the  availability  of  all  needed  data.  The  final 
data  requirements  for  each  integrated  test  are 
subsequently  captured  in  an  Integrated  Test  Plan  (ITP) 
that  serves  as  the  basis  for  individual  Detailed  Test 
Plans  (DTPs)  developed  by  the  test  executors. 

Data  collected  in  accordance  with  these  require¬ 
ments  is  then  used  by  the  System  Engineer  to  support 
analyses  that  generate  estimates  of  the  current  level  of 
system  performance  relative  to  the  Cl  SRD  required 
levels.  The  confidence  in  these  performance  estimates 
improves  as  the  capability,  fidelity,  and  maturity  of  the 
element  representations  tested  increase. 

Simulation  in  NMD  Test  Program 

Importance  of  Simulation  to  the  Test  Program 

The  importance  of  simulation  and  HWIL  testing 
comes  from  their  repeatability,  controllability,  and  cost 
effectiveness.  The  NMD  System,  because  of  its  unique 
nature,  does  not  lend  itself  readily  to  comprehensive 
field  tests  and  demonstrations.  Testing  is  necessarily 
limited  by  the  impracticality  of  fully  duplicating  the 
threat  and  the  tactical  environment.  As  a  result,  system 
performance  evaluation  and  requirements  definition 
must  rely  on  simulation.  Simulation  provides  a  link  to 
connect  the  performance  of  interceptor  elements  with 
the  NMD  System  performance  requirements.  Top-level 
requirements,  operating  concepts,  and  scenarios  of  bat¬ 
tle  action  are  generated  at  the  system  level  and  are  car¬ 
ried  down  in  greater  levels  of  detail  and  allocation  to 


the  next  level  of  detail.  Highly  detailed  models  of  pro¬ 
posed  technologies  are  carried  upward  to  performance 
function  models,  which  are  compatible  with  and  sup¬ 
portive  of  the  element  models. 

Contribution  of  Simulation  to  Test  and  Evaluation 

Simulation  has  contributed  to  T&E  activities  in 
monitoring  system  requirements,  extending  the  bound¬ 
ary  envelope  of  the  system  under  evaluation,  collecting 
data  for  model  validation,  and  providing  data  for  analy¬ 
ses  to  support  deployment  decisions.  Simulations  in  the 
test  program  support  three  major  areas:  1)  system 
requirements  analysis,  2)  design  performance  analysis, 
and  3)  flight  test  analysis. 

System  requirements  analysis  involves  system 
effectiveness  studies  for  different  scenarios  and  one-on- 
one  threat  analysis.  Topics  include  the  force 
operational  placement  effectiveness  and  netted  dis¬ 
tributed  replicated  architecture  trades.  Also  included 
are  verification  of  Single-Shot  Kill  Probability,  weapon 
target  pairing,  surveillance  management,  kill  assess¬ 
ment,  discrimination  and  threat  typing,  and,  in  general, 
battle  management. 

Design  performance  analysis  serves  to  verify  ele¬ 
ment  B-specification  requirements,  verify  BMC3  per¬ 
formance,  and  provide  subsystem  performance  budget 
analysis  and  coverage  analysis.  System  statistical  per¬ 
formance  consists  of  Monte  Carlo  analysis  to  show  sta¬ 
tistics  on  IFTUs,  kill  vehicle  propellant  remaining,  miss 
distance,  and  a  host  of  other  measures-of-effectiveness 
parameters.  This  leads  to  an  evaluation  of  subsystem 
Single-Shot  Engagement  Kill  Probability  (SSEKP)  per¬ 
formance  evaluation  and  an  evaluation  of  system-level 
drivers,  such  as  radar  and  launcher  saturation. 

The  focus  of  data  analysis  from  flight  tests  using 
simulation  will  be,  primarily,  to  show  compliance  with 
the  element  and  interface  specifications.  The  goal  is  to 
meet  the  specific  flight  test  objectives.  Requirement 
flow-down  started  from  test  objectives  and  sub¬ 
objectives  designated  for  each  individual  test. 
Traceability  must  extend  to  the  system  level  test  and 
data  requirements  as  well  as  Developmental  Test  and 
Evaluation  objectives.  Figures,  tables,  and  discussions 
in  an  analysis  report  must  clearly  demonstrate  this 
traceability. 

Simulations  are  used  to  assess  element  functional 
performance  in  specific  areas.  For  radar  and  infrared 
sensors,  the  analysis  may  include  tracking  position  and 
velocity  accuracy,  launch  and  intercept  point  estimation 
accuracy,  threat  discrimination,  and  kill  assessment 
analysis.  For  interceptors,  assessment  may  include 
seeker  signature  and  data  processing,  guidance,  naviga- 
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tion  and  control,  aerodynamics,  telemetry,  in-flight 
guidance  update,  and  threat  object  map  processing.  For 
the  BMC3  element,  the  functional  analysis  will  extend 
to  cover  threat  evaluations,  sensor  and  track  manage¬ 
ment,  weapon  management  and  assignment,  engage¬ 
ment  control,  HIC,  discrimination,  kill  assessment,  and 
command,  control,  and  communications.  At  the  system 
level,  simulations  support  the  assessment  of  protection 
effectiveness,  event  timelines,  kill  assessment,  system 
interfaces,  and  operational  connectivity.  Additional 
uses  for  simulations  are  found  in  test  scenario  develop¬ 
ment,  target  and  threat  trajectory  and  signature  analysis, 
and  lethality  analysis.  Simulations  are  used  to  extend 
from  one-on-one  and  few-on-few  real  life  flight  test 
scenarios  to  many-on-many  system  effectiveness 
analyses. 

A  number  of  simulations  are  used  for  threat  sce¬ 
nario  generation,  weapon  system  engagement,  and 
high-fidelity  system  simulation,  including  BMC3, 
communications,  radar,  and  missile.  BMDO  has 
accredited  software  to  support  terminal  endgame 
lethality  simulation,  which  models  body-to-body  and 
fragment  kill  of  ballistic  missile  payloads  from  kinetic 
energy  weapon  attack.  There  are  also  scenario  genera¬ 
tors  to  model  many  objects,  including  threat  target  re¬ 
entry  vehicles,  boosters,  fragments,  and  interceptors. 

Relationship  of  Simulation  to  Integrated  Ground  and 
Flight  Testing 

Simulations  are  conducted  for  mission  planning 
and  for  predicting  the  IFTs’  results.  The  HWIL  IGTs 


are  run  to  provide  confidence  in  flight  test  execution 
and  in  deriving  detailed  data  requirements.  After  a 
flight  test  has  been  conducted,  the  assessment  is  com¬ 
pleted  with  the  release  of  a  test  execution  report  and  a 
Post  Test  Analysis  Report,  and  data  is  subsequently 
released  to  the  BMD  community.  Data  from  flight  tests 
also  serves  to  validate  the  models  used  in  simulations 
and  HWIL  tests.  The  validated  models  serve  to  extend 
the  performance  boundary  from  few-on-few  to  many- 
on-many  threat  and  interceptor  scenarios/trajectories. 
The  pre-test  and  post-test  relationships  to  the  flight  test 
are  shown  in  Figure  3.  Simulation  and  HWIL  testing 
provide  high  confidence  in  system  performance.  This 
confidence  is  increased  as  IGTs  are  validated  by  flight 
test  data  comparative  analyses  as  the  NMD  test  pro¬ 
gram  progresses. 

Ground  Testing  in  NMD  Test  Program 

Importance  of  IGTs  to  the  Test  Program 

An  NMD  IGT  is  a  type  of  test  involving  one  or 
more  NMD  element  test  articles  operating  in  a  system 
context  in  a  nondestructive  mode.  A  test  article  consists 
of  the  actual  element  operational  software  (SWIL) 
operating  within  the  actual  data  processing  hardware 
(HWIL). 

IGTs  provide  the  opportunity  for  system  elements 
to  interact  by  physical  or  phenomenological 
stimulation.  The  primary  roles  of  IGTs  are  to: 

•  Provide  IFT  pre-mission/risk  reduction  checkout 

•  Validate  functional  interfaces  between  elements 


Use  Simulation(s)  to  PREDICT 
OUTCOME  OF  HWIL  AND  FLIGHT 
TESTS  Using  Required  Parameters 
for  Comparisons 


iterate  Test  Process  • 
Until  Simulation(s)  and 
HWIL  Results  Agree 


HARDWARE 
IN  THE  LOOP 


Use  HWILtoP/?£D/C7 
OUTCOME  OF  FLIGHT  TESTS 
Using  the  Required  Parameters 
^  for  Comparisons 


Use  HWIL  to  ANALYZE  FLIGHT 
TEST  RESULTS  to  Determine 
Reasons  for  Differences  Between 
Predictions  and  Observations 


Use  Simulation(s)  to 
ANALYZE  HWIL  AND 
FLIGHT  TEST  RESULTS 
to  Determine  Reasons 
for  Differences  Between 
Predictions  and  Observations 


MODIFY  AND  VALIDATE 
AfW/Z.  According  to  Flight 
Test  Results 


MODIFY  AND  VALIDATE 
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to  Test  Results.  Only  Validated 
Simulations  Can  Extend  the 
Envelope  of  Credibility. 


Figure  3.  Pre  and  Post  Test  Relationships 


UNCLASSIFIED 


UNaASSIFIED 


•  Subject  system  to  stressing  environments  and 
scenarios 

•  Evaluate  system  capabilities  over  tactical  battlespace 

•  Test  in  a  controlled  and  repeatable  environment 

•  Test  in  a  nondestructive  system  mode 

•  Provide  data  for  NMD  System  performance 
evaluation 

•  Provide  an  environment  for  operational  analysis  of 
system  maturity. 

IGTs  generate  comparative  IFT  test  data  to  validate 
a  portion  of  the  NMD  performance  envelope  repre¬ 
sented  by  each  test  configuration.  The  IGTs  are  used  to 
extend  the  scope  of  IFTs,  evaluate  system  performance 
at  the  edges  of  the  engagement  envelope,  and  combine 
hardware  and  software  development  in  realistic  system 
configurations,  IGTs  are  repetitive  in  nature.  This  pro¬ 
vides  the  tester  with  a  method  to  collect  sufficient  data 
to  gain  confidence  in  the  system. 

Contribution  of  IGTs  to  Test  and  Evaluation 

IGTs  benefit  the  NMD  test  program  through  testing 
the  evolving  system  configuration  using  simulation/ 
emulation  and  real  data  processing  hardware/software. 
HIC,  simulated  real-world  threats,  global  and  local 
environments,  and  multiple  elements  interact.  IGTs 
provide  valid  and  useful  test  data,  a  wider  range  for 
testing  at  a  far  lower  cost  than  a  comparable  IFT,  and 
an  opportunity  to  analyze  “what-if  *  criteria  for  mean¬ 
ingful  assessments  of  element  capabilities  when  merged 
into  a  system.  IGTs,  distinct  from  IFTs,  are  standalone 
tests,  capable  of  providing  multi-run  test  data. 

Relationship  of  IGTs  to  Simulation  and  Integrated 
Flight  Testing 

Used  as  pre-mission  risk  reduction  measures,  IGTs 
provide  confidence  in  IFT  execution  by  predicting  ele¬ 
ment  performance,  exercising  interelement  interfaces 
and  message  exchanges,  and  affirming  the  ability  to 
provide  the  required  data.  Ground  testing  reduces  the 
uncertainties  associated  with  flight  tests  to  only  those 
that  must  be  resolved  through  actual  flights.  IGTs  are 
used  in  a  post-mission  mode  to  assess  achievement  of 
IFT  objectives  and  to  investigate  anomalies. 

IGTs  are  used  to  evaluate  the  performance  of  the 
evolving  objective  system  in  operational  scenarios  that 
cannot  be  tested  in  IFTs  due  to  constraints  such  as 
engagement  geometry,  multiple  object  attack  scenarios, 
and  nuclear  environments.  IGTs  provide  a  wider  range 
of  testing  at  lower  cost  than  IFTs,  yet  retain  realistic 
element  inter-operation  with  human  operator  participa¬ 
tion. 

The  primary  NMD  System-level  ground  test  tool  is 


the  Integrated  System  Test  Capability  (ISTC),  located 
at  the  Advanced  Research  Center  (ARC).  It  is  a  com¬ 
puter-based  system  for  testing  actual  NMD  element 
data  processors  (HWIL)  and  software  (SWIL)  in  an 
integrated  configuration  through  the  use  of  a  common 
test  environment.  The  ISTC’s  function  is  to  test  func¬ 
tionality,  reliability,  operability,  and  interoperability  of 
embedded  computers  within  integrated  elements  of  the 
NMD  System.  Individual  elements  of  the  NMD 
System  are  represented  in  the  ISTC  on  computer 
stations,  known  as  nodes.  Each  node  incorporates 
actual  element  mission  and  communications  processors, 
running  actual  element  software.  A  system 
communications  network  driven  by  real-time  system 
interfaces  interconnects  the  individual  element  nodes 
and  threat  and  environment  input  data.  Separate  ISTC 
networks  controlling  the  test  configuration  and 
providing  the  environmental  data  also  connect  the 
nodes.  The  ISTC  infrastructure  supplies  each 
autonomous  node  with  a  simulated  threat  and 
associated  environments,  natural  and  man-made,  which 
are  consistent  for  each  NMD  element  in  the  test 
scenario.  In  this  manner,  ISTC  exercises  the  entire 
NMD  System  simultaneously. 

The  embedded  element  processors  are  the  NMD 
“tactical  hardware”  being  tested  in  the  ISTC.  The  ISTC 
interfaces  directly  with  the  actual  BMC3,  so  that  system 
level  operability  issues  can  be  addressed.  It  accurately 
represents  actual  tactical  communications  messages  in 
terms  of  timing,  format,  and  content,  allowing  testers  to 
resolve  connectivity  and  interoperability  questions. 

The  primary  roles  of  ISTC  are  to: 

•  Support  the  evaluation  of  the  system  performance  of 

the  NMD  System: 

-  Test  a  broad  range  of  threat  scenarios  with  multiple 
engagements 

-  Test  multiple  sensor  operations  such  as  correlation 
and  discrimination 

-  Test  system  recovery  from  failures  and  casualties 

•  Determine  the  interoperability  and  reliability  of  the 

deployed  software  in  the  NMD  System: 

Verify  the  NMD  System  interfaces 

-  Conduct  extended  tests  to  determine  software 
reliability 

-  Drive  the  integrated  NMD  in  all  of  its  modes  of 
operation 

•  Exercise  HIC  to  the  extent  planned  in  the  NMD 

System: 

-  Exercise  actual  command  centers  and  operators  in 
the  loop  with  stressing  scenarios. 

•  Extend  the  envelope  of  flight  testing: 

-  The  size  of  the  threat 

-  The  types  of  threats 
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-  Complex  and  multiple  engagements 

-  NMD  System  operation  in  the  presence  of  the 
full  range  of  natural  and  hostile  environments 

•  Support  Integrated  Right  Tests: 

-  Reduce  the  risk  of  conducting  a  flight  test 

-  Determine  the  test  article  readiness  by  exercising  it 
in  a  system  environment 

-  Verify  the  test  plan  by  simulating  the  test  and 
displaying  the  expected  results 

-  Confirm  test  integration  by  stimulating  the  test 
assets  to  verify  readiness. 

-  Support  post-mission  analysis 

-  Integrate  all  recorded  data  and  reconstruct  the  test 

-  Provide  representations  for  missing  elements. 

Risk  Reduction  Right  fRRF)  Testing  in  the  NMD  Test 
Program 

Importance  of  RRFs  to  the  Test  Program 

RRFs  (formerly  called  targets  of  opportunity)  are 
important  to  the  NMD  test  program  for  two  reasons: 
cost  and  schedule.  Right  testing  becomes  increasingly 
expensive  as  element  representations  mature  and  as  the 
test  support  instrumentation  augments  each  test.  There¬ 
fore,  test  configurations,  test  integration,  procedures, 
and  control  are  more  complex  with  each  successive 
IFT.  An  RRF  maximizes  the  use  of  all  the  parts  avail¬ 
able  for  the  IFT  with  the  exception  of  two  high-dollar 
items,  the  interceptor  and  the  NMD  validated  target. 
Because  the  RRF  is  nondestructive,  most  of  the  IFT 
end-to-end  connectivity  can  be  checked  out  for  a  rela¬ 
tively  fractional  cost  of  an  IFT.  Secondarily,  with  a  3 
plus  3  deployment-readiness  fast-track  schedule,  each 
test  must  be  supportive  of  the  program  development  or 
deployment  schedule.  Due  to  the  key  role  of  the  flight 
test  program,  RRFs  expand  the  operational  test  oppor¬ 
tunities.  Additional  flights  are  planned  to  support  ele¬ 
ment  and  system  level  risk  reduction.  RRFs  are 
scheduled  to  complete  the  development  of  the  NMD 
test  infrastructure  and,  in  some  cases,  supplement  the 
element  development  tests.  To  this  end,  RRFs  serve  as 
an  efficient  component  of  the  NMD  test  program  to 
maximize  NMD  dollars  and  augment  the  planned  inte¬ 
grated  ground  and  flight  tests. 

Contribution  of  RRFs  to  Test  and  Evaluation 

The  contribution  of  RRFs  to  the  NMD  test  program 
can  be  recognized  in  their  use  as  an  element  or  system 
tool.  RRF  events  provide  results  on  sensors,  BMC3, 
communications  connectivity,  and  test  infrastructure 
that  is  valuable  to  overall  risk  mitigation  and  system 


evaluation.  The  use  of  RRFs  is  specifically  identified 
in  the  SEP  evaluation  plans.  These  flight  test  resources, 
coupled  with  IGTs  and  model  and  simulation  analyses, 
provide  a  full  spectrum  of  tools  to  validate,  verify, 
accredit,  and  evaluate  NMD  System  performance  in 
support  of  the  NMD  Deployment  Readiness  Review 
(DRR).  Although  this  paper  focuses  on  system-level 
testing,  an  RRF  may  be  utilized  by  a  single  element  or 
in  a  context  of  less  than  a  complete  system.  For  exam¬ 
ple,  a  sensor  element  may  require  an  RRF  to  check  out 
functional  or  physical  interoperability,  to  perform  sen¬ 
sor  analysis,  or  to  verify  other  element  requirements. 
There  are  various  ways  that  an  RRF  can  address  and/or 
resolve  test  or  program  needs  for  the  system  level.  The 
following  describes  some  uses:  1)  perform  assessments 
and  quick  look  analysis  of  the  IFT  test  conduct,  2)  per¬ 
form  post  test  analysis  addressing  objectives  for  the 
following  IFT,  3)  address  T&E  data  nominal/anominal 
requirements,  4)  address  “holes”  in  the  IFT  results. 
RRFs  have  flexible  requirements  and  objectives  that 
can  be  tailored  to  specific  needs  for  element  or  system 
test  risk  reduction. 

Relationship  of  RRFs  to  Simulation  and  Integrated 
Ground  and  Flight  Testing 

Inherent  in  the  current  planned  NMD  test  program 
are  RRFs,  which  provide  a  cost  effective  approach  to 
reduce  program  risk.  RRFs  have  been  successfully 
used  in  the  past  to  support  development  objectives.  The 
RRF  test  program  leverages  off  non-NMD  existing 
flight  tests  or  other  separate  targets  of  opportunity 
where  available  NMD  element  representations  are  run 
on-line.  Essentially,  the  RRF  test  configuration  “piggy¬ 
backs”  off  targets  of  opportunity  such  as  a  Multi- 
Service  Launch  System  (MSLS)  or  a  ballistic  missile 
Follow-On  Test  and  Evaluation  (FOT&E)  flight  test. 
The  target  flight  opportunity  allows  the  evolving  NMD 
test  architecture  to  participate  in  an  “associate  opera¬ 
tions”  role  with  the  target  test  program’s  office  of  pri¬ 
mary  responsibility.  Through  this  relationship,  an  RRF 
provides  an  operational  environment  by  which  each 
element,  in  a  system  configuration  context,  examines 
interoperability  with  other  elements  and  the  external 
systems.  Target  flight  schedules  are  assessed  to  allow 
RRFs  to  be  conducted  prior  to  IFTs.  An  RRF  provides 
a  vehicle  to  exercise  and  test  IFT  configuration,  verify 
uprange  element  integration  prior  to  the  IFT  mission, 
and  verify  real-time  test  procedures.  An  RRF  supports 
pre-mission  testing  by  providing  both  recorded  and  live 
flight  test  data.  This  data  is  used  to  drive  the  HWIL 
IGT  test  setups  to  reduce  risk. 
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Flight  Testing  in  the  NMD  Test  Program 

Importance  of  IFTs  to  the  Test  Program 

Flight  tests  verify  NMD  System  performance  and 
provide  fully  integrated  end-to-end  system  demonstra¬ 
tions  in  the  real  world  using  threat  representative  tar¬ 
gets.  IFTs  also  demonstrate  end-to-end  target  detection, 
acquisition,  tracking,  correlation,  and  hand-over  per¬ 
formance;  demonstrate  real-time  discrimination  per¬ 
formance;  and  demonstrate  NMD  System  kill 
assessment  capability.  In  addition,  flight  tests  demon¬ 
strate  the  ability  of  the  NMD  System  to  develop  and 
coordinate  battle  management  plans;  prepare,  launch, 
and  fly  out  a  designated  weapon,  and  kill  a  threat  repre¬ 
sentative  target.  They  also  demonstrate  integration, 
interface  compatibility,  and  performance  of  the  NMD 
System,  subsystem  hardware  and  software,  and  HIC 
operations  and  present  a  strong  contribution  to  NMD 
System  effectiveness  and  suitability. 

Contribution  of  IFTs  to  Test  and  Evaluation 

Flight  tests  provide  data  to  verify  that  system 
requirements  and  performance  objectives  have  been 
met.  They  allow  an  incremental  development  process 
with  a  test-as-you-go  philosophy.  This  provides  assur¬ 
ance  of  system  maturity  prior  to  moving  into  the  next 
development  phase  and  prior  to  authorization  of  subse¬ 
quent  development  and  deployment  expenditures. 
Flight  tests  replicate,  with  an  increasing  degree  of 
accuracy,  the  actual  operational  use  of  the  system. 
They  allow  the  actual  operators  to  have  hands-on 
experience  prior  to  actual  deployment  and  combat  use. 

Flight  tests  demonstrate  that  errors  can  be  zeroed  in 
performing  Hit-to-Kill  (HTK)  intercepts  and  provide 
data  to  allow  for  optimum  allocation  of  response  time 
to  verify  speed,  propulsion,  capability,  and  maneu¬ 
vering.  Flight  tests  also  demonstrate  that  integration 
and  interface  requirements  have  been  satisfied,  and  that 
the  interactive  effects  of  the  interceptor,  target,  NMD 
elements,  and  environment  variables  are  accounted  for 
in  a  flight  environment.  In  addition,  flight  tests  con¬ 
tribute  to  the  verification  of  lethality,  probability  of  hit, 
probability  of  kill,  target  debris,  and  mission  reliability. 

Relationship  of  IFTs  to  Simulation  and  Integrated 
Ground  Testing 

A  complex  system  like  NMD  cannot  be  thoroughly 
flight  tested  throughout  the  performance  envelope 
because  of  test  limitations  and  cost.  Each  flight  test 
costs  in  excess  of  $50M,  which  severely  limits  the 
number  that  can  be  conducted.  At  the  test  range,  a  tac¬ 
tical  site  cannot  be  replicated  because  of  physical  con¬ 


straints,  and  range  safety  governs  the  selection  of 
intercept  points.  Therefore,  a  practical  approach  must 
be  conceived  to  supplement  flight  tests  with  ground 
tests.  Historically  this  has  been  accomplished  by 
simulating  the  system,  validating  the  simulations  with 
flight  test  data,  and  then  using  the  simulations  for 
extended  performance  and  statistical  testing.  Using  a 
combination  of  HWIL,  SWIL,  and  simulations,  ground 
tests  are  used  to  test  the  total  engagement  space  and 
threat  spectrum.  They  assess  the  functional  interfaces 
between  the  elements,  subject  the  system  to  stressing 
scenarios  such  as  many  on  many,  and  evaluate  target- 
intercept  boundary  conditions.  IGTs  are  conducted  in 
controlled  and  repeatable  environments  in  a  non¬ 
destructive  mode.  They  help  to  identify  “unknown 
unknowns”  in  an  interactive  system  context,  and  they 
verify  interoperability  of  NMD  System  elements 
throughout  the  system’s  performance  envelope.  Flight 
tests  cannot  do  this.  Series  of  IGTs  are  conducted  with 
tactical  configurations  (representations,  simulations  and 
HWIL/SWIL)  to  generate  management  decision  data. 
In  addition  to  ground  tests,  simulations  provide  repre¬ 
sentations  of  elements  that  are  not  yet  mature  enough 
for  the  flight  test  program  and  representations  of  com¬ 
plex  environments,  helping  to  overcome  limitations  in 
actual  flight  testing.  Simulations  verify  interface 
requirements  and  software  integration,  validate  algo¬ 
rithm  integration,  and  reduce  element  integration  risk. 
Simulations  will  be  employed  to  effectively  repeat  data 
points  in  order  to  improve  the  statistical  sample  or  to 
determine  overlooked  or  directly  unmeasured 
parameters.  Flight  test  data  validates  ground  test 
design,  results,  HWIL,  and  simulation  fidelity. 

Evaluation  of  Test  Results  and  System  Performance 

This  section  describes  the  analysis  relationships 
specifically  among  IFTs,  IGTs,  and  system  perform¬ 
ance  evaluation.  Common  features  between  IFTs  and 
IGTs,  the  process  for  analyzing  the  features,  and  exter¬ 
nal  document  support  are  also  discussed. 

Purpose 

The  purpose  of  post  test  analysis  is  to  verify  that 
the  NMD  Program  T&E  objectives  have  been  attained 
and  data  collection  requirements  have  been  met.  The 
resulting  data  is  verified  against  T&E  objectives  that 
are  agreed  upon  at  the  test  design  review.  The  System 
Engineer  can  then  use  this  data  to  assess  progress 
toward  required  system  performance.  T&E  objectives 
are  derived  from  two  sources:  the  Integrated  Test  Plan, 
which  provides  test  objectives  flowed  from  the  TEMP 
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for  scheduled  tests  to  be  conducted  during  the  specified 
fiscal  year  test  program,  and  the  System  Requirements 
Document,  which  provides  detailed  test  and  analysis 
requirements. 

Relationships 

Ground  tests  and  flight  tests  are  analyzed  by  com¬ 
paring  the  test  results  against  the  test  objectives.  Suc¬ 
cess  and  increased  confidence  for  each  test  result  begin 
with  a  commitment  to  early  involvement  in  all  test 
activities.  Pre-mission  activities  provide  interaction 
with  test  operators,  thereby  providing  opportunities  to 
refine  analysis  procedures,  influence  test  execution,  and 
optimize  the  test  results.  Involvement  continues  with 
engineering  judgment  of  the  cause  of  and  estimates  of 
the  accuracy  of  each  test  result,  the  presentation  and 
reporting  of  the  findings,  and  interaction  with  the  sys¬ 
tem  performance  evaluation  process.  System 
performance  evaluation  is  the  process  of  determining 
progress  toward  attainment  of  system  performance 
requirements.  This  process  involves  the  evaluation  of 
data  from  system-  and  element-level  models  and  simu¬ 
lations,  analyses,  demonstrations,  and  inspections.  Post 
test  analysis  supports  the  System  Engineer  in  this 
activity  by  the  submittal  of  individual  test  results  in  a 
manner  such  that  they  may  be  used  to  rebalance  test 
objectives,  performance  requirements,  and  future  tests. 

IFT/IGT  Commonalities 

Although  IGTs  and  IFTs  hold  unique  attributes,  a 
common  thread  exists  in  the  post  test  analysis  of  the  test 
results.  IFT  and  IGT  commonalities  include:  mission 
profiles,  timelines,  and  message  traffic.  For  each  of 
these  test  categories,  data  sets  collected  as  truth, 
planned,  and  observed  data  are  compared  as  a  signifi¬ 
cant  part  of  the  analysis.  Figure  4  displays  how  truth 
data  is  compared  with  observed,  planned  is  compared 
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Figure  4.  Data  Comparisons 


with  observed,  and  truth  is  compared  with  planned. 
Truth  data  is  defined  as  data  that  characterizes  the 
actual  occurrence  of  an  event  with  respect  to  a  pre¬ 
defined  reference  coordinate  system.  Examples  of  truth 
data  include  DSP  and  MSLS  source  data.  Planned  data 
is  defined  as  a-priori  data  that  anticipates  the  occur¬ 
rence  of  an  event  constrained  by  a  limited  set  of 
assumptions.  Examples  of  planned  data  include 
simulated  and  pre-mission  source  data.  Observed  data 
is  defined  as  a-posteriori  data  that  is  measured  or  cal¬ 
culated  as  a  result  of  the  occurrence  of  an  event  and  is 
constrained  only  by  environmental  and  test  architecture 
conditions.  Examples  of  observed  data  include  position 
and  velocity  state  vectors. 

The  time  sequence  of  events  is  also  assessed  within 
the  recorded  data.  Interactions  among  element 
representations  are  evaluated.  Engineering  anomalies 
and  statistical  estimates  of  the  causes  with  respect  to 
individual  tests  are  assessed  and  presented  in  a  format 
for  the  System  Engineer  to  conduct  system  performance 
evaluation.  In  this  framework  all  test  data  is  statisti¬ 
cally  and  comparatively  analyzed,  identifying  potential 
program  shortcomings  requiring  corrective  measures. 

Analysis  Process 

Typically  the  post  test  analysis  process  consists  of 
activities  in  four  major  areas:  data  preparation,  data 
qualification,  data  analysis,  and  post-test  analysis 
reporting.  These  activities  are  sequential,  supporting 
each  subsequent  step  in  the  process  shown  in  Figure  5. 
Data  preparation  consists  of  conversion  of  binary  (raw) 
data  to  ASCII  (text)  data  and  template  formatting  for 
data  base  management.  Data  qualification  consists  of 
management  of  censored,  truncated,  missing,  or  outlier 
data.  Data  analysis  reduces  all  test  data  to  meaningful 
results  with  respect  to  the  test  objectives.  Post  test 
analysis  reporting  publishes  and  distributes  the  findings 
through  hardcopy  and  electronic  media.  Specific  analy¬ 
sis  is  performed  in  the  following  areas:  coverage  analy¬ 
sis,  timeline  analysis,  and  message  analysis.  Coverage 
analysis  spans  NMD  defense  coverage  of  each  of  the  50 
United  States.  Timeline  analysis  measures  the  occur¬ 
rence  and  response  of  critical  NMD  events.  Message 
analysis  tracks  the  source  and  destination  of  NMD  mes¬ 
sage  contents  that  stimulate  and  respond  to  coverage 
functions  and  timeline  parameters. 

The  post  test  analysis  process  provides  statistical 
and  comparative  analysis  of  truth,  planned,  and  observ¬ 
able  test  data.  Message  contents,  timelines,  critical  test 
parameters,  and  other  data  are  analyzed  to  produce 
measurement  error  and  specification  deviations.  Error 
and  deviation  output  is  cost-effectively  generated  with 
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Figure  5.  Post  Test  Analysis  Process 


commercial-off-the-shelf  (COTS)  software  resident  on 
a  network  of  Windows-  and  Unix-based  platforms. 
This  value-added  configuration  eliminates  the  need  for 
expensive  and  time-consuming  new  development  of  the 
analytic  tools  and  decreases  technical  risk  by  utilizing 
industry-standard  COTS  software. 

Routine  outputs  from  the  COTS  software  (tabular, 
histograms,  scattergrams,  distribution  plots,  etc.)  are 
collectively  assembled,  correlated,  and  interpreted  as 
specific  test  results  from  each  individual  IFT  and  IGT 
test  series.  Industry-standard  word  processing  COTS 
software  completes  the  process  of  publishing  the  test 
results  as  interim  and  Final  Post  Test  Analysis  Reports 
(PTARs). 

Each  PTAR  aggressively  supports  the  system  engi¬ 
neering  evaluation  process  with  the  inclusion  of  step- 
by-step  achievements  of  each  specific  analysis  for  the 
test  objectives.  These  contents  and  the  analyzed  data 
are  used  by  the  System  Engineer  to  assess  the  attain¬ 
ment  of  system  performance  requirements  and  are  fur¬ 
ther  used  to  rebalance  test  objectives,  performance 
requirements,  and  future  tests. 

Document  Support 

Post  test  analysis  aggressively  supports  and 
reviews  the  spectrum  of  compliance  documents 
(TEMP,  SRD,  SEP,  DTP,  etc.).  These  documents  drive 
the  IFT  and  IGT  developmental  tests,  test  objectives, 
and  data  requirements  that  produce  the  recorded  data. 
The  data  is  then  assessed  according  to  the  SEP 
requirements  such  that  the  system  engineering  evalua¬ 
tion  process  advances  towards  successful  program 
completion. 

The  source  for  all  collected  data  (truth,  planned, 
observed)  is  identified  in  the  appropriate  Data  Man¬ 
agement  Plan.  The  conditions  under  which  the  data  is 
gathered  and  the  format  to  be  used  for  test-objective 


data  presentations  are  then  developed.  These  presenta¬ 
tions  may  take  the  form  of  tables,  graphs,  curves,  bar 
charts,  and  others.  This  decomposition  is  executed  for 
all  IFTs  and  IGTs. 

Progress  in  achieving  DT&E  program  objectives, 
threat  assessment,  and  demonstrated  maturity  of  the 
NMD  System  are  key  milestones  for  the  NMD  Deploy¬ 
ment  Readiness  Review.  IFT  and  IGT  test  objectives 
represent  a  subset  of  the  DT&E  objectives.  Source 
documents  (ITP,  SRD,  etc.)  contain  the  objectives  and 
are  decomposed  into  sub-objectives  and  data  require¬ 
ments  for  each  decomposition.  From  this  process, 
measurable  parameters  are  extracted  whose  evaluations 
are  ultimately  presented  as  evidence  of  attainment 
toward  the  NMD  System  objective. 

Test  Challenge  Accepted  and  Pursued  TSummarv) 

Testing  a  system  as  complicated  and  vital  as  the 
NMD  System  is  challenging,  but  not  formidable.  The 
test  program  discussed  herein  implements  processes 
developed  for  allocating  test  requirements,  executing 
testing,  and  performing  test  analysis  and  evaluation. 
Further,  system  technical  performance  measures  and 
test  objectives  are  addressed  in  progressive  incremental 
integration  of  elements  in  test  configurations  for  FY97- 
FYOO.  The  NMD  System  capability  is  demonstrated 
through  verification  of  interface  compatibility,  interop¬ 
erability,  and  integrated  performance  of  the  elements. 
The  NMD  test  program  focuses  on  deployment  readi¬ 
ness  through  complementary  integrated  flight  testing, 
integrated  ground  tests,  simulation,  and  risk  reduction 
flight  tests.  A  Lead  System  Integrator  has  been  selected 
and  given  the  authority  to  proceed  to  execute  a  test  pro¬ 
gram  that  provides  increasing  confidence  that  we  will 
meet  the  NMD  challenge  and  be  ready  for  a  deploy¬ 
ment  decision  in  FYOO. 
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